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FOREWORD 


This  research  and  development  was  conducted  in  response  to  Navy  Decision  Coordina¬ 
ting  Paper,  NDCP-Z0828-PN,  Performance  Aids  Test  and  Evaluation,  under  the  sponsor¬ 
ship  of  the  Deputy  Chief  of  Naval  Operations  (OP -01).  This  work  seeks  to  further  the 
objectives  of  the  NDCP,  which  are:  to  define  the  state-of-the-art  in  3ob  Performance 
Aid  (3PA)  technology;  to  develop  a  conceptual  model  for  an  integrated  3P  A- based 
personnel  system,  including  cost  benefits  and  trade-off  analysis;  to  test  the  3PA  concept; 
and  to  quantify  performance  increments  and  cost  benefits  obtainable  from  various 
applications. 

This  report  compares  the  effectiveness  of  the  state  tables  and  ordnance  publications 
as  troubleshooting  aids. 

Sincere  appreciation  is  expressed  to  the  personnel  of  the  Combat  Systems  Technical 
Schools  Command  at  Mare  Island,  California,  for  their  assistance  in  conducting  this  study. 
GMMC  K.  Randall  deserves  a  special  note  of  appreciation  for  sharing  his  expertise  and 
spending  many  12-hour  days  monitoring  the  evaluation.  Appreciation  is  extended  to  CW04 
3.  Davis  and  FTCM  3.  Fuller  who  contributed  their  time  during  the  planning  phase  of  this 
study. 


3AMES  F.  KELLY,  3R 
Commanding  Officer 


3AMES  3.  REGAN 
Technical  Director 


SUMMARY 


Problem 


Binary  digital  circuit  elements  alternate  between  two  states,  which  are  often 
referred  to  as  "1"  and  "0"  or  as  "high"  and  "low."  When  troubleshooting  digital  equipment, 
it  is  not  enough  to  know  whether  the  output  of  a  circuit  element  is  high  or  low— it  is 
necessary  to  know  whether  the  output  is  high  or  low  at  specific  points  in  time.  Numerous 
job  performance  aids  PPAs)  have  been  developed  to  assist  technicians  in  troubleshooting 
electronic  equipment,  but  none  of  those  in  current  use  address  the  problem  of  timing  in 
digital  systems.  A  recently  developed  3PA,  the  state  table,  provides  technicians  with 
time-related  troubleshooting  information,  but  the  concept  had  not  been  evaluated  by  the 
Navy. 

Objective 

The  objective  of  this  research  was  to  compare  state  tables  with  an  existing  form  of 
troubleshooting  documentation,  the  ordnance  publication  (OP). 

Approach 

Eight  Navy  technicians,  after  receiving  instructions  in  the  use  of  state  tables,  were 
directed  to  troubleshoot  12  problems  on  the  Missile  Control  Unit  (MCU)  of  the  NATO 
Seasparrow  Surface  Missile  System  (NSSMS).  Each  subject  used  state  tables  for  six 
problems  and  OPs  for  six  others.  Problems  of  three  levels  of  difficulty  were  used:  easy, 
medium,  and  hard.  The  subjects  were  given  a  maximum  time  limit  on  each  problem:  30 
minutes  for  easy  problems,  45  for  medium,  and  60  for  hard.  The  performance  of  each 
subject  was  evaluated  and  compared  on  the  basis  of  success  rate,  time  required,  the 
number  of  checks  required,  and  the  number  of  valid  checks  performed.  (A  check  was 
considered  valid  if  it  was  (1)  prescribed  by  the  troubleshooting  document  and  (2)  correctly 
interpreted  by  the  subject.)  By  questionnaire,  the  subjects  were  then  asked  to  rate  and 
compare  state  tables  and  OPs. 

Findings 

Using  state  tables,  the  subjects  solved  31  problems  in  25  hours.  With  OPs  they  solved 
only  nine  problems,  and  took  33.8  hours  to  do  so.  A  nonparametric  test,  the  Wilcoxon 
Matched-Pairs  Signed-Ranks  Test,  indicated  that  both  the  increase  in  the  number  of 
problems  solved  and  the  decrease  in  troubleshooting  time  were  associated  with  a  single 
factor— use  of  state  tables. 

With  state  tables,  710  checks  were  made,  with  606  of  them  valid.  With  OPs,  1404 
checks  were  made,  with  1062  valid. 

Seven  of  the  eight  subjects  rated  the  state  tables  as  excellent  or  good  for  all  problem 
types.  OPs  were  rated  as  good  or  adequate  by  seven  subjects  for  easy  problems,  by  six  for 
medium  problems,  and  by  only  three  for  hard  problems.  All  eight  subjects  preferred  state 
tables  over  OPs  for  medium  and  difficult  problems. 

Conclusions 


The  results  of  this  research  indicate  that  state  tables  are  effective  3PAs  for 
troubleshooting  digital  equipment  in  the  NSSMS.  The  state  tables,  supported  by  a 
troubleshooting  guide,  enabled  technicians  to  isolate  over  three  times  the  number  of 
faults  in  25  percent  less  time. 
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Recommendations 


1.  The  Navy  should  incorporate  the  state  tables  and  troubleshooting  guide  into 
NSSMS  documentation. 

2.  The  Navy  should  provide  potential  users  of  state  tables  with  adequate  training, 
including  hands-on  experience. 

3.  The  state  table  technique  should  also  be  compared  with  the  troubleshooting 
techniques  contained  in  conventional  technical  manuals,  in  functionally-oriented 
maintenance  manuals,  and  in  other  documents  that  have  been  prepared  to  facilitate  the 
troubleshooting  of  digital  electronic  equipment. 

4.  Extension  of  this  comparison  to  include  fleet  personnel  would  provide  a  better 
understanding  of  the  positive  and  negative  aspects  of  using  state  tables  for  troubleshoot¬ 
ing. 
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INTRODUCTION 

Problem 


Binary  digital  circuit  elements  alternate  between  two  states,  which  are  often 
referred  to  as  "l"  and  ‘W  or  as  "high"  and  "low."  When  troubleshooting  digital  equipment, 
it  is  not  enough  to  know  whether  the  output  of  a  circuit  element  is  high  or  low--it  is 
necessary  to  know  whether  the  output  is  high  or  low  at  specific  points  in  time.  Numerous 
job  performance  aids  (JPAs)  have  been  developed  to  assist  technicians  in  troubleshooting 
electronic  equipment,  but  none  of  those  in  current  use  address  the  unique  problem  of 
timing  in  digital  systems.  A  recently  developed  3PA,  the  state  table,  provides  technicians 
with  time-related  troubleshooting  information  but  the  concept  had  not  been  evaluated  by 
the  Navy. 

Purpose 


The  objective  of  this  research  was  to  compare  state  tables  with  an  existing  form  of 
troubleshooting  documentation,  the  ordnance  publication  (OP). 

Background 

Maintenance  of  a  system  requires  both  planned  and  unplanned  maintenance.  Planned 
maintenance  (e.g.,  cleaning,  inspecting,  adjusting,  replacing,  etc.)  is  performed  on  a 
periodic  basis  to  ensure  continued  operation  of  the  system.  These  tasks  are  generally  well 
defined,  with  specific  start  and  end  points,  and  the  time  required  to  perform  each  task  is 
fairly  predictable.  Unplanned  maintenance  actions  G.e.,  those  needed  to  repair  a 
malfunctioning  system)  include  troubleshooting  and  removal  and  replacement  actions. 

Troubleshooting  is  a  deductive  process  that  requires  the  technician  to  make  a  series 
of  logical  decisions  based  upon  observation  or  measurement  of  the  states  of  various 
circuit  elements.  Conventional  technical  manials,  functionally-oriented  maintenance 
manuals  (FOMMs),  and  other  3PAs  have  been  designed  to  assist  technicians  in  making 
these  decisions  quickly  and  correctly,  but  troubleshooting  remains  the  most  difficult 
maintenance  task.  This  is  especially  true  in  digital  equipment  and  systems. 

Problems  in  Digital  Systems  Maintenance 

In  analog  circuits,  each  signal  follows  a  single  path,  though  it  may  occasionally  loop 
or  branch  out.  The  technician  can  usually  trace  the  signal  from  its  origin  to  its  output  and 
isolate  a  problem  to  a  faulty  circuit  element  by  a  process  of  elimination. 

It  is  seldom  possible  to  use  such  techniques  when  troubleshooting  digital  systems.  In 
a  digital  circuit,  a  given  circuit  element  may  serve  different  purposes  at  different  times, 
and  digital  data  may  flow  through  different  paths  from  one  instant  to  the  next.  Because 
of  the  complex  branching  in  digital  systems,  a  fault  may  have  wide-ranging  effects  that 
do  not  appear,  initially,  to  have  any  connection  with  each  other  or  with  the  faulty  circuit 
element. 

Digital  circuit  elements  switch  back  and  forth  between  two  states  and  being  in  the 
wrong  state,  or  between  states,  constitutes  an  error.  The  proper  state  of  each  element  is 
a  f  motion  of  time.  An  error  may  or  may  not  be  detected  at  all,  depending  upon  the  mode 
the  system  is  in  and  the  timing  of  events  G.e.,  an  error  may  be  undetectable  if  the  state 
of  the  malfunctioning  component  does  not  matter  during  the  time  the  system  is  being 
observed,  a  distinct  possibility  with  digital  systems). 
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For  example,  assume  that  the  output  of  a  certain  flip-flop  is  supposed  to  be  high  only 
when  data  are  entering  the  system  (flip-flops  are  common  digital  circuit  elements).  A 
failure  that  caused  this  flip-flop's  output  to  be  high  at  all  times  would  be  detected  only  if 
a  test  was  made  when  data  were  not  entering  the  system. 

Built-in  Test  Equipment  (BITE)  and  Diagnostic  Programs 

To  compensate  for  their  complexity,  digital  systems  are  often  designed  with  built-in 
test  equipment  (BITE)  and  diagnostic  programs  that  perform,  automatically,  many  of  the 
complex  tests  required  to  isolate  digital  equipment  malfutctions.  BITE  and  diagnostics 
are  expensive  and  their  comprehensiveness  is  limited  by  their  cost.  For  example,  if  a 
diagnostic  has  been  developed  to  isolate  90  percent  of  all  faults,  the  cost  of  increasing 
fault  isolation  by  1  or  2  percent  could  double  the  cost  of  the  diagnostic  (Porta,  1979). 
Regardless  of  cost,  however,  even  the  most  comprehensive  diagnostics  cannot  isolate  all 
faults  in  a  system  and  some  percentage  of  the  troubleshooting  must  be  done  by  more  or 
less  well- trained  and  experienced  technicians. 

Currently  Available  3ob  Performance  Aids  (3PAs) 

In  addition  to  BITE  and  diagnostics,  various  JPA  formats  have  been  developed  to  help 
the  technician  troubleshoot  digital  equipment.  One  approach  complements  the  diagnostics 
by  furnishing  functional  block  diagrams  that  lead  the  troubleshooter  to  a  suspected  group 
of  components  where  the  fault  may  be  located.  This  approach  also  indicates  additional 
fault  groups  that  the  technician  can  test  if  the  fault  is  not  found  in  the  initially  identified 
group. 

Functionally  oriented  maintenance  manuals  (FOMMs)  have  been  adapted  to  digital 
systems  by  the  addition  of  timing  diagrams,  but  the  timing  information  has  not  been 
integrated  with  logic-flow  information.  The  technician  is  therefore  required  to  mentally 
integrate  the  two  types  of  information  while  troubleshooting. 

The  increasing  use  of  digital  components  has  made  the  troubleshooting  approach  of 
current  3PAs  generally  inadequate.  When  3PAs  were  developed  for  analog  equipment,  all 
fault  symptoms  were  identified  and  corrective  procedures  were  developed  for  each 
symptom  or  set  of  symptoms.  But  since  digital  components  serve  multiple  functions,  and 
since  digital  signal  paths  are  seldom  straightline  functions,  an  almost  infinite  number  of 
symptoms  are  possible  in  a  large  digital  system.  The  technician  who  troubleshoots  digital 
systems  needs  a  complete  functional  understanding  of  the  system,  usually  attained  after 
many  years  of  training  and  experience.  A  recently  developed  3PA,  the  state  table,  may 
make  it  possible  for  less  experienced  technicians  to  troubleshoot  digital  systems. 


DESCRIPTION  OF  THE  STATE  TABLE  CONCEPT 

State  tables  provide  concise,  uncomplicated,  and  time-related  representations  of  the 
factions  of  digital  equipment  and  systems.  They  not  only  describe  the  sequence  of 
events  required  to  execute  each  function,  they  also  show  the  logic  signal  combinations 
that  characterize  each  event.  State  tables  are  system-oriented,  thus  enabling  technicians 
to  approach  troubleshooting  tasks  from  the  standpoint  of  system  characteristics  rather 
than  equipment  characteristics.  Within  each  state  table,  normal  system  characteristics 
are  correlated  with  the  logic  states  of  selected  circuit  elements  in  a  way  that  indicates 
where  one  should  look  for  the  fault. 
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Figure  1  illustrates  the  state  table  format.  The  circuit  elements  are  listed  across  the 
top  and  the  states  of  the  elements  are  represented  by  zeros  (low  state)  and  ones  (high 
state).  The  events  enabled  by  specific  states  of  the  circuit  elements  are  listed  on  the 
right  side.  For  example,  the  event  SYNC  i  RECEIVED  is  enabled  when  1-A-l  is  high  (1) 
and  1-C-l  is  low  (0). 


Figure  1.  State  Table  (typical). 


To  use  a  state  table  successfully,  the  technician  must  understand  the  various 
functions  of  the  equipment  and  be  able  to  recognize  malfunctions.  As  an  aid, 
troubleshooting  guides  have  been  developed  to  correlate  BITE  indications  with  the  state 
tables.  Typically,  the  troubleshooting  guide  and  state  tables  are  used  together  as  follows: 

1.  The  technician  initiates  a  built-in  test  and  observes  which  fault  indicators  are 
illuminated. 

2.  The  technician  uses  the  troubleshooting  guide  to  determine  the  area  where  the 
fault  is  most  likely  to  be. 

3.  The  guide  directs  the  technician  to  the  state  table  that  covers  the  area  indicated 
in  step  2. 


4.  The  technician  notes  which  functions  are  good  and  which  are  not,  and  uses  the 
state  table  to  isolate  the  fault  to  a  single  circuit  element. 

5.  The  technician  compares  the  actual  states  of  the  suspect  circuit  elements  with 
the  states  prescribed  for  them  in  the  state  table  to  verify  their  condition. 

Figure  2  presents  an  example  of  how  a  state  table  can  be  used  to  isolate  a  defective 
circuit  element.  In  this  example,  a  preliminary  investigation  indicated  that  functions  2,  3, 
and  6  were  good  and  that  1,  4,  and  5  were  faulty.  The  state  table  shows  that  only  one 
circuit  component,  C-5,  is  related  to  all  three  faulty  functions. 


Figure  2.  State  table  example. 


During  the  analysis  that  precedes  state  table  development,  the  system  is  described 
function  by  function  and  the  state  tables  are  developed  accordingly.  The  malfunction 
indications  of  the  system  are  then  considered  and  integrated  into  the  troubleshooting 
guide. 

Thurmond  and  Booher  (1979),  in  comparing  the  adequacy  and  cost  of  preparation  of 
state  tables  with  other  documentation,  found  that  state  tables  offered  the  following 
advantages: 

1.  Complete  digital  systems  content  coverage. 

2.  Ease  of  use. 

3.  Improved  display  of  information. 

4.  Improved  troubleshooting  training. 

5.  Reduced  skill  and  knowledge  requirements. 
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APPROACH 


Subjects 

This  evaluation  should  be  considered  a  pilot  test  because  only  eight  subjects  were 
tested  during  the  2-week  evaluation  period.  The  relatively  small  N  was  aue  to  the 
difficulty  of  obtaining  subjects  with  the  necessary  training  and  experience  on  the  Missile 
Control  Unit  (MCU)  of  the  MK-29  NATO  Seasparrow  Surface  Missile  System  (NSSMS). 

Four  of  the  subjects  were  students  who  had  just  graduated  from  the  MK  29  NATO 
Launcher  School  at  the  Combat  Systems  Technical  Schools  Command  (CSTSC),  Mare 
Island,  California.  Three  of  the  students  had  attended  Basic  Electricity  and  Electronics 
School,  Gunners  Mate  "A”  School,  and  the  MK  29  NATO  Launcher  School.  The  fourth 
student  had  attended  seven  schools  before  attending  the  MK  29  NATO  Launcher  School. 
The  students  had  been  in  the  Navy  from  2  to  3  years. 

The  fifth  and  sixth  subjects  were  MK  29  NATO  Launcher  instructors;  the  seventh  was 
a  NATO  Fire  Control  System  instructor;  and  the  eighth  was  a  Fire  Control  Technician 
awaiting  assignment  to  a  school.  The  instructors  had  been  in  the  Navy  from  8  to  14  years 
and  the  Fire  Control  Technician  had  been  in  for  6  years.  The  last  two  mentioned  subjects 
had  not  received  any  training  on  the  MCU  of  the  NSSMS. 

Equipment 

The  MCU  of  the  NSSMS  was  selected  for  use  in  this  evaluation  because  it  has  a  high 
failure  rate  and  the  existing  documentation  did  not  appear  to  be  adequate  for  technical 
support.  The  state  tables  and  troubleshooting  guide  for  the  MCU  were  developed  by 
Hughes  Aircraft  Company,  Fullerton. 

Procedure 


Each  subject  was  given  3ft  days  of  instruction  in  state  table  usage,  followed  by  6  days 
of  performance  testing.  A  partial  repeated  measures  design  was  used  in  which  each 
subject  attempted  to  solve  12  problems  in  four  separate  sessions.  Each  subject  was  tested 
individually,  and  each  session  lasted  approximately  2ft  hours. 

During  the  testing,  subjects  were  divided  into  two  groups,  each  consisting  of  two 
MK  29  students,  one  MK  29  instructor,  and  either  the  Fire  Control  System  Instructor  or 
the  Fire  Control  Technician.  The  order  in  which  each  group  used  the  troubleshooting 
methods  was  varied:  One  group  used  state  tables  to  solve  problems  during  their  first  and 
last  sessions,  and  OPs  during  their  second  and  third  sessions;  the  other  group  used  OPs  in 
their  first  and  last  sessions,  and  state  tables  in  their  second  and  third  sessions.  Testing 
sessions  were  held  in  alternating  sequence,  a  session  with  one  group  being  followed  by  a 
session  with  the  other. 

Each  session  consisted  of  three  problems:  one  easy,  one  medium,  and  one  hard.  The 
problems  were  casualties  drawn  from  a  master  casualty  file  developed  for  MCU 
troubleshooting  training  by  CSTSC.  The  casualty  file  was  developed  before  state  tables 
were  introduced  to  CSTSC.  The  levels  of  problem  difficulty  were  defined  as: 

1.  Easy  (E)~a  casualty  with  an  instruction  cycle  of  2-3  instructions  or  3-4  control 
signals. 
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2.  Medium  (MV- a  casualty  with  an  instruction  cycle  of  4-6  instructions  or  6-10 
control  signals. 

3.  Hard  (HV-a  closed  loop  casualty  that  could  have  an  unlimited  number  of 
instruction  cycles  or  control  signals.  These  were  CROM  addressing  faults  and  DUS  8 
instruction  cycle  faults.  (CROM  and  DUS  8  are  computer  component  designations.) 

Easy  problems  had  a  30  minute  time  limit;  medium,  45  minutes;  and  hard  problems, 
60  minutes.  There  were  four  similar  problems  for  each  level  of  difficulty,  and  the  order 
of  presenting  the  problems  was  varied  (counter  balanced)  to  eliminate  order  effects.  The 
independent  triable  of  interest  was  the  type  of  troubleshooting  document  (OP  or  state 
table)  used  by  the  subject.  The  experimental  design  is  given  in  Table  1. 


Table  1 

Experimental  Design 


Type  of 

Troubleshooting 

Problem 

State 

Tables 

Number  of  Observations 
(Problems  Attempted) 

Ordnance 

Publications 

Easy 

16 

16 

Medium 

16 

16 

Hard 

16 

16 

Note.  The  number  of  observations  was  obtained  by  having  each  of  the 
eight  subjects  solve  two  problems  in  each  cell. 


Data 

The  following  dependent  measures  were  determined: 

1.  Number  of  problems  (E,  M,  or  H)  solved  within  the  time  limits. 

2.  Amount  of  troubleshooting  time. 

3.  Total  number  of  checks  (both  valid  and  invalid). 

4.  Number  of  checks  that  were  valid. 

Valid  checks  were  those  checks  that  were  (1)  prescribed  by  the  troubleshooting 
document  and  (2)  interpreted  correctly  by  the  technician.  Invalid  checks  were  those  that 
were  not  prescribed  by  the  documented  procedures,  or  prescribed  checks  that  were 
misinterpreted. 

QtAlitative  data  were  collected  by  a  questionnaire  that  required  subjects  to  rate 
state  tables  and  OPs  for  each  level  of  difficulty,  and  to  compare  state  tables  with  OPs. 
(The  questionnaire  is  shown  in  Figure  3.) 
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Subject  10 


Session 


A.  What  is  your  reaction  to  (state  tables/OPs)  for  problem: 


Excellent 

Good 

Adequate 

Less  than 

Adequate 

Poor 

( 

) 

(  ) 

( 

) 

( 

) 

(  ) 

( 

) 

(  ) 

( 

) 

( 

) 

(  ) 

( 

) 

(  ) 

( 

) 

( 

) 

(  ) 

8.  Were  the  (state  tables/OPs)  easy  to  follow  for  problem: 


C.  Were  the  (state  tables/OPs)  helpful  in  locating  faults  for  problem: 

Ves  No 

1  (  )  (  ) 

2  (  )  (  ) 

3  (  )  (  ) 

D.  Which  technique  did  you  prefer  for  each  problem? 


State 

Tables 

OPs 

Both 

( 

) 

( 

) 

( 

) 

( 

) 

( 

) 

( 

) 

( 

) 

( 

) 

( 

) 

E.  Were  the  state  tables  an  improvement  over  OPs  on  your  ability  to  detect 
and  troubleshoot  for  each  problem? 


solving  than  the  OPs? 


Yes 

No 

1 

( 

) 

(  ) 

2 

( 

) 

(  ) 

3 

( 

) 

(  ) 

rovi  de 

a  more 

logical 

sequence  to 

follow  for 

Yes 

No 

1 

( 

) 

(  ) 

2 

( 

) 

(  ) 

3 

( 

) 

(  ) 

1  in  solving 

probl ems 

■> 

State 

Tables 

OPf 

Nei  t‘ier 

( 

) 

(  ) 

(  ) 

( 

) 

(  ) 

(  ) 

( 

) 

(  ) 

(  ) 
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RESULTS 


Since  none  of  the  subjects  had  used  state  tables  in  an  operational  setting,  the  first 
problem  (the  easy  one)  of  the  first  session  served  as  a  practice  problem  and  the  subjects 
were  given  constructive  feedback  on  their  procedures.  The  problem  was  repeated  during 
the  second  session  with  state  tables.  The  data  from  the  repeated  problem  were  included 
in  the  results  to  keep  the  cell  sizes  equal.  (The  MCU  is  complex  enough  that  there  is 
little  change  in  the  likelihood  of  solving  a  problem  repeated  only  once.)  All  subjects  had 
used  OPs  in  operational  settings,  so  no  OP  practice  problems  were  included. 

With  state  tables,  the  eight  subjects  solved  31  of  48  problems  in  25  hours.  With  OPs, 
they  solved  9  of  48  problems  in  33.8  hours.  The  means  and  standard  deviations  are  given 
in  Tahle  2. 


Table  2 

Problems  Completed  and  Troubleshooting  Time 


Problems  Completed 
per  Subject 

Troubleshooting  Timea 
(Minutes) 

OP 

State  Tables 

OP 

State  Table 

Mean 

1.125 

3.875 

253.375 

187.375 

Standard 

Deviation 

1.246 

1.458 

24.951 

36.265 

^ime  per  subject  spent  on  all  12  problems  attempted. 


A  nonparametric  test,  the  Wilcoxon  Matchec-Pairs  Signed-Ranks  Test  (Roscoe,  1975), 
was  used  to  test  the  null  hypothesis  that  state  tables  and  OPs  lead  to  no  difference  in 
troubleshooting  time  or  number  of  completed  problems.  The  hypothesis  was  rejected, 
with  all  differences  in  the  same  direction  (a  =  .01).  Thus,  both  the  increase  in  the  number 
of  problems  completed  correctly  and  the  decrease  in  the  troubleshooting  time  are 
attributable  to  a  single  factor— the  use  of  state  tables. 

When  the  evaluation  results  were  correlated  to  problem  type,  it  was  discovered  that 
the  level  of  difficulty  had  different  effects  on  the  time  required  to  solve  problems  with 
state  tables  and  OPs,  as  well  as  varying  effects  on  the  average  number  of  problems  solved 
by  each  subject.  These  differences  are  summarized  in  Figures  4  and  5. 

Additional  measures  included  total  checks  and  the  number  of  those  checks  that  were 
valid.  With  state  tables,  710  checks  were  made,  of  which  606  (85%)  were  valid.  With 
OPs,  1444  checks  were  made,  of  which  1062  (74%)  were  valid. 

The  questionnaire  provided  qualitative  data.  Table  3  presents  the  ratings  given  by 
the  eight  subjects  for  each  problem  type  and  documentation  type.  Table  4  summarizes 
their  documentation  preferences  for  each  level  of  problem  difficulty. 
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Figure  4.  Number  of  problems  solved  (by  problem  type)  by  eight 
subjects. 
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Figure  5.  Troubleshooting  time  taken  by  subjects 
when  solving  easy,  medium,  and  hard 
problems. 
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Number  of  Subjects  Rating  Documentation  as; 


Problem 

Type 

Excellent 

Good 

Adequate 

Less  than 
Adequate 

Poor 

State  Tables 

Easy 

5 

2 

0 

1 

0 

Medium 

5 

2 

1 

0 

0 

Hard 

6 

1 

0 

1 

0 

OPs 

Easy 

0 

3 

4 

1 

0 

Medium 

0 

4 

2 

0 

2 

Hard 

0 

1 

2 

1 

4 

Note.  N  a  8  for  all  cells. 


Table  4 

Preference  Rating  by  Problem  Type 


Problem 

Type 

Number  of  Subjects  who  Preferred: 

State  Tables 

OPs 

No  Preference 

Easy 

6 

0 

2 

Medium 

8 

0 

0 

Hard 

8 

0 

0 

Note.  N  a  8  for  all  cells. 
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DISCUSSION  AND  CONCLUSIONS 


Although  this  evaluation  was  a  pilot  test  with  the  noted  constraints,  the  evidence 
supports  the  conclusion  that  state  tables  are  effective  troubleshooting  JPAs  for  digital 
systems.  When  compared  with  existing  documentation  for  the  NSSMS,  the  state  tables 
enabled  users  to  isolate  more  faults.  The  mean  number  of  problems  completed  correctly 
by  each  subject  was  3.875  when  state  tables  were  used;  it  was  only  1.125  when  OPs  were 
used.  In  addition,  with  state  tables  the  troubleshooting  time  for  the  12  problems  was 
reduced  by  more  than  25  percent  (2027  minutes  overall  for  OPs  versus  1499  minutes  for 
state  tables). 

With  state  tables  there  was  more  variability  in  the  subject's  troubleshooting  times. 
The  standard  deviation  for  state  tables  was  36.265,  while  for  OPs  it  was  only  24.951. 
There  are  two  reasons  for  this  variabiHty.  First,  when  OPs  were  used,  troubleshooting 
times  approached  the  time  limits  for  ail  problem  types,  thus  reducing  the  overall  range  of 
times.  With  state  tables,  however,  some  subjects  solved  the  easy  and  medium  problems  in 
extremely  short  periods  of  time,,  while  spending  a  relatively  long  time  on  hard  problems. 
For  example,  the  medium  dif  M  „ulty  problem  in  the  second  session  with  OPs  required  the 
full  time  limit,  45  minutes,  for  all  subjects;  with  state  tables  the  mean  was  only  8.25 
minutes.  At  the  same  time,  tte  nr  ean  for  the  hard  problem  in  the  second  session  with  OPs 
was  58.125  minutes,  whik  witn  state  tables,  the  mean  was  49.50.  The  time  with  OPs 
varied  only  about  13  minutes,  but  with  state  tables  it  varied  over  40  minutes. 

The  second  reason  cor  the  variability  in  troubleshooting  times  with  state  tables  was 
the  newness  of  the  technique  and  the  experience  level  of  the  subjects.  Experienced 
technicians  (i.e.,  the  instructors)  had  to  "relearn"  troubleshooting  with  the  state  table 
technique,  which  could  have  produced  interference  on  some  problems.  Also,  the 
inexperienced  technicians  (i.e.,  the  recent  graduates)  may  have  found  it  easier  to  apply 
the  state  table  technique  in  some  areas  and  not  in  others. 

When  problem  difficulty  is  correlated  with  the  type  of  documentation,  the  results  do 
not  appear  to  be  consistent  for  state  tables.  However,  the  classification  of  problems  as 
easy,  medium,  and  hard  was  based  on  previous  experience  with  OPs,  not  state  tables. 
When  examining  the  data  for  OPs,  the  number  of  problems  solved  and  the  troubleshooting 
time  are  consistent  with  the  problem  types.  For  state  tables,  however,  the  medium 
problem  type  does  not  appear  to  be  appropriately  classified.  It  may  be  an  anomaly 
associated  with  one  particular  problem;  that  is,  state  table  logic  may  be  exceptionally 
superior  to  OP  logic  for  solving  that  problem. 

Almost  twice  as  many  checks  were  required  when  using  OPs  as  were  required  when 
using  state  tables.  The  large  difference  in  total  checks  is  reflected  in  the  number  of  valid 
checks:  There  were  400  more  valid  checks  with  OPs  than  with  state  tables.  It  appears 
that  state  tables  provide  an  efficient  means  of  fault  isolation.  With  fewer  checks,  and 
with  more  of  them  being  valid,  a  troubleshooter  can  reduce  equipment  down  time 
significantly. 

The  relative  superiority  of  state  tables  over  OPs  is  enhanced  when  the  users' 
subjective  ratings  and  comparisons  are  included.  Seven  of  the  eight  subjects  rated  the 
state  tables  as  excellent  or  good  for  all  problem  types.  OPs,  on  the  other  hand,  were 
rated  as  good  or  adequate  by  seven  subjects  for  easy  problems,  by  six  for  medium 
problems,  and  by  only  three  for  hard  problems.  Finally,  all  eight  subjects  preferred  state 
tables  over  OPs  for  medium  and  difficult  problems,  and  six  preferred  them  to  OPs  for 
easy  problems. 
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RECOMMEN  DATIONS 


Based  on  the  results  of  this  pilot  test,  it  is  recommended  that  the  Navy  incorporate 
the  state  tables  and  troubleshooting  guide  as  part  of  the  present  NSSMS  documentation. 
If  this  recommendation  is  accepted,  the  state  tables  and  troubleshooting  guide  will  need 
to  be  given  some  type  of  introduction  (i.e.,  the  potential  users  should  have  hands-on 
training  with  state  tables.)  Failure  to  provide  an  introductory  framework  for  the  state 
table  technique  could  lead  to  misinterpretation  and  rejection  by  the  fleet. 

Extension  of  this  comparison  to  include  other  equipment  and  more  personnel  would 
provide  additional  information  on  the  positive  and  negative  aspects  of  using  state  tables 
for  troubleshooting. 

A  more  comprehensive  evaluation  should  be  conducted.  The  state  table  technique 
should  be  compared  to  conventional  technical  manuals,  FOMMs,  and  other  3PAs  that  have 
been  prepared  to  facilitate  troubleshooting  digital  electronic  equipment. 
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